Aiming at the problems of poor stability and accuracy of feature point detection in traditional image matching algorithms, a new image feature point matching algorithm based on Scale-invariant Center surround Filter Detection (SCFD) was proposed. Firstly, a multi-scale space was constructed, a center surround filter was used to detect feature points of a image at different scales, Harris method and sub-pixel interpolation were applied to acquire the stable feature points. Secondly, Oriented fast and Rotated Binary Robust Independent Elementary Feature (BRIEF) (ORB) algorithm was combined to confirm the main direction of feature points and construct the description operator of feature points. Finally, Hamming distance was used to complete the matching, Least Median Squares (LMeds) theorem and Maximum Likelihood (ML) estimation were used to eliminate wrong matching points. The experimental results show that, the matching precision of the proposed algorithm is up to 96.6%, which is 2 times of that of the ORB algorithm when the scale changes. The running time of the proposed algorithm is 19.8% of that of Scale Invariant Feature Transform (SIFT) and 28.3% of that of Speed-Up Robust Feature (SURF). The proposed algorithm can effectively improve the stability and accuracy of feature point detection, and has better matching effects under the circumstances of different angle of view, scale scaling, rotation change and brightness variation.